{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Remaining Useful Life (advanced)\n",
    "<p style=\"margin:30px\">\n",
    "    <img style=\"display:inline; margin-right:50px\" width=50% src=\"https://www.featuretools.com/wp-content/uploads/2017/12/FeatureLabs-Logo-Tangerine-800.png\" alt=\"Featuretools\" />\n",
    "    <img style=\"display:inline\" width=15% src=\"https://upload.wikimedia.org/wikipedia/commons/e/e5/NASA_logo.svg\" alt=\"NASA\" />\n",
    "</p>\n",
    "\n",
    "This notebook has a more advanced workflow than [the other notebook](Simple%20Featuretools%20RUL%20Demo.ipynb) for predicting Remaining Useful Life (RUL). If you are a new to either this dataset or Featuretools, I would recommend reading the other notebook first. \n",
    "\n",
    "## Highlights\n",
    "* Demonstrate how novel entityset structures improve predictive accuracy\n",
    "* Build custom primitives using time-series functions from [tsfresh](https://github.com/blue-yonder/tsfresh)\n",
    "* Improve Mean Absolute Error by tuning hyper parameters with [BTB](https://github.com/HDI-Project/BTB)\n",
    "\n",
    "Here is a collection of mean absolute errors from both notebooks. Though we've used averages where possible (denoted by \\*), the randomness in the Random Forest Regressor and how we choose labels from the train data changes the score.\n",
    "\n",
    "|                                 | Train/Validation MAE|  Test MAE|\n",
    "|---------------------------------|--------------------------------|\n",
    "| Median Baseline                 | 72.06*              | 50.66*   |\n",
    "| Simple Featuretools             | 40.92*              | 39.56    |\n",
    "| Advanced: Custom Primitives     | 35.90*              | 28.84    |\n",
    "| Advanced: Hyperparameter Tuning | 34.80*              | 27.85    |\n",
    "\n",
    "\n",
    "# Step 1: Load Data\n",
    "We load in the train data using the same function we used in the previous notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data with:\n",
      "61249 Recordings\n",
      "249 Engines\n",
      "21 Sensor Measurements\n",
      "3 Operational Settings\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>engine_no</th>\n",
       "      <th>time_in_cycles</th>\n",
       "      <th>operational_setting_1</th>\n",
       "      <th>operational_setting_2</th>\n",
       "      <th>operational_setting_3</th>\n",
       "      <th>sensor_measurement_1</th>\n",
       "      <th>sensor_measurement_2</th>\n",
       "      <th>sensor_measurement_3</th>\n",
       "      <th>sensor_measurement_4</th>\n",
       "      <th>sensor_measurement_5</th>\n",
       "      <th>...</th>\n",
       "      <th>sensor_measurement_14</th>\n",
       "      <th>sensor_measurement_15</th>\n",
       "      <th>sensor_measurement_16</th>\n",
       "      <th>sensor_measurement_17</th>\n",
       "      <th>sensor_measurement_18</th>\n",
       "      <th>sensor_measurement_19</th>\n",
       "      <th>sensor_measurement_20</th>\n",
       "      <th>sensor_measurement_21</th>\n",
       "      <th>index</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>42.0049</td>\n",
       "      <td>0.8400</td>\n",
       "      <td>100.0</td>\n",
       "      <td>445.00</td>\n",
       "      <td>549.68</td>\n",
       "      <td>1343.43</td>\n",
       "      <td>1112.93</td>\n",
       "      <td>3.91</td>\n",
       "      <td>...</td>\n",
       "      <td>8074.83</td>\n",
       "      <td>9.3335</td>\n",
       "      <td>0.02</td>\n",
       "      <td>330</td>\n",
       "      <td>2212</td>\n",
       "      <td>100.00</td>\n",
       "      <td>10.62</td>\n",
       "      <td>6.3670</td>\n",
       "      <td>0</td>\n",
       "      <td>2000-01-01 00:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>20.0020</td>\n",
       "      <td>0.7002</td>\n",
       "      <td>100.0</td>\n",
       "      <td>491.19</td>\n",
       "      <td>606.07</td>\n",
       "      <td>1477.61</td>\n",
       "      <td>1237.50</td>\n",
       "      <td>9.35</td>\n",
       "      <td>...</td>\n",
       "      <td>8046.13</td>\n",
       "      <td>9.1913</td>\n",
       "      <td>0.02</td>\n",
       "      <td>361</td>\n",
       "      <td>2324</td>\n",
       "      <td>100.00</td>\n",
       "      <td>24.37</td>\n",
       "      <td>14.6552</td>\n",
       "      <td>1</td>\n",
       "      <td>2000-01-01 00:10:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>42.0038</td>\n",
       "      <td>0.8409</td>\n",
       "      <td>100.0</td>\n",
       "      <td>445.00</td>\n",
       "      <td>548.95</td>\n",
       "      <td>1343.12</td>\n",
       "      <td>1117.05</td>\n",
       "      <td>3.91</td>\n",
       "      <td>...</td>\n",
       "      <td>8066.62</td>\n",
       "      <td>9.4007</td>\n",
       "      <td>0.02</td>\n",
       "      <td>329</td>\n",
       "      <td>2212</td>\n",
       "      <td>100.00</td>\n",
       "      <td>10.48</td>\n",
       "      <td>6.4213</td>\n",
       "      <td>2</td>\n",
       "      <td>2000-01-01 00:20:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>42.0000</td>\n",
       "      <td>0.8400</td>\n",
       "      <td>100.0</td>\n",
       "      <td>445.00</td>\n",
       "      <td>548.70</td>\n",
       "      <td>1341.24</td>\n",
       "      <td>1118.03</td>\n",
       "      <td>3.91</td>\n",
       "      <td>...</td>\n",
       "      <td>8076.05</td>\n",
       "      <td>9.3369</td>\n",
       "      <td>0.02</td>\n",
       "      <td>328</td>\n",
       "      <td>2212</td>\n",
       "      <td>100.00</td>\n",
       "      <td>10.54</td>\n",
       "      <td>6.4176</td>\n",
       "      <td>3</td>\n",
       "      <td>2000-01-01 00:30:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>25.0063</td>\n",
       "      <td>0.6207</td>\n",
       "      <td>60.0</td>\n",
       "      <td>462.54</td>\n",
       "      <td>536.10</td>\n",
       "      <td>1255.23</td>\n",
       "      <td>1033.59</td>\n",
       "      <td>7.05</td>\n",
       "      <td>...</td>\n",
       "      <td>7865.80</td>\n",
       "      <td>10.8366</td>\n",
       "      <td>0.02</td>\n",
       "      <td>305</td>\n",
       "      <td>1915</td>\n",
       "      <td>84.93</td>\n",
       "      <td>14.03</td>\n",
       "      <td>8.6754</td>\n",
       "      <td>4</td>\n",
       "      <td>2000-01-01 00:40:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       engine_no  time_in_cycles  operational_setting_1  \\\n",
       "index                                                     \n",
       "0              1               1                42.0049   \n",
       "1              1               2                20.0020   \n",
       "2              1               3                42.0038   \n",
       "3              1               4                42.0000   \n",
       "4              1               5                25.0063   \n",
       "\n",
       "       operational_setting_2  operational_setting_3  sensor_measurement_1  \\\n",
       "index                                                                       \n",
       "0                     0.8400                  100.0                445.00   \n",
       "1                     0.7002                  100.0                491.19   \n",
       "2                     0.8409                  100.0                445.00   \n",
       "3                     0.8400                  100.0                445.00   \n",
       "4                     0.6207                   60.0                462.54   \n",
       "\n",
       "       sensor_measurement_2  sensor_measurement_3  sensor_measurement_4  \\\n",
       "index                                                                     \n",
       "0                    549.68               1343.43               1112.93   \n",
       "1                    606.07               1477.61               1237.50   \n",
       "2                    548.95               1343.12               1117.05   \n",
       "3                    548.70               1341.24               1118.03   \n",
       "4                    536.10               1255.23               1033.59   \n",
       "\n",
       "       sensor_measurement_5         ...          sensor_measurement_14  \\\n",
       "index                               ...                                  \n",
       "0                      3.91         ...                        8074.83   \n",
       "1                      9.35         ...                        8046.13   \n",
       "2                      3.91         ...                        8066.62   \n",
       "3                      3.91         ...                        8076.05   \n",
       "4                      7.05         ...                        7865.80   \n",
       "\n",
       "       sensor_measurement_15  sensor_measurement_16  sensor_measurement_17  \\\n",
       "index                                                                        \n",
       "0                     9.3335                   0.02                    330   \n",
       "1                     9.1913                   0.02                    361   \n",
       "2                     9.4007                   0.02                    329   \n",
       "3                     9.3369                   0.02                    328   \n",
       "4                    10.8366                   0.02                    305   \n",
       "\n",
       "       sensor_measurement_18  sensor_measurement_19  sensor_measurement_20  \\\n",
       "index                                                                        \n",
       "0                       2212                 100.00                  10.62   \n",
       "1                       2324                 100.00                  24.37   \n",
       "2                       2212                 100.00                  10.48   \n",
       "3                       2212                 100.00                  10.54   \n",
       "4                       1915                  84.93                  14.03   \n",
       "\n",
       "       sensor_measurement_21  index                time  \n",
       "index                                                    \n",
       "0                     6.3670      0 2000-01-01 00:00:00  \n",
       "1                    14.6552      1 2000-01-01 00:10:00  \n",
       "2                     6.4213      2 2000-01-01 00:20:00  \n",
       "3                     6.4176      3 2000-01-01 00:30:00  \n",
       "4                     8.6754      4 2000-01-01 00:40:00  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import featuretools as ft\n",
    "import utils\n",
    "\n",
    "data_path = 'data/train_FD004.txt'\n",
    "data = utils.load_data(data_path)\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also make cutoff times by selecting a random cutoff time from the life of each engine. We're going to make 5 sets of cutoff times to use for cross validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5/5 [00:02<00:00,  2.30it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>engine_no</th>\n",
       "      <th>cutoff_time</th>\n",
       "      <th>RUL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2000-01-02 18:20:00</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2000-01-04 19:30:00</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>2000-01-05 09:20:00</td>\n",
       "      <td>294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>2000-01-08 22:40:00</td>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>2000-01-10 15:00:00</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       engine_no         cutoff_time  RUL\n",
       "index                                    \n",
       "1              1 2000-01-02 18:20:00   66\n",
       "2              2 2000-01-04 19:30:00   70\n",
       "3              3 2000-01-05 09:20:00  294\n",
       "4              4 2000-01-08 22:40:00   56\n",
       "5              5 2000-01-10 15:00:00    7"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "splits = 5\n",
    "cutoff_time_list = []\n",
    "\n",
    "for i in tqdm(range(splits)):\n",
    "    cutoff_time_list.append(utils.make_cutoff_times(data))\n",
    "\n",
    "cutoff_time_list[0].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to do something fancy for our entityset. The values for `operational_setting` 1-3 are continuous but create an implicit relation between different engines. If two engines have a similar `operational_setting`, it could indicate that we should expect the sensor measurements to mean similar things. We make clusters of those settings using [KMeans](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html) from scikit-learn and make a new entity from the clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Entityset: Dataset\n",
       "  Entities:\n",
       "    recordings [Rows: 61249, Columns: 29]\n",
       "    engines [Rows: 249, Columns: 2]\n",
       "    settings_clusters [Rows: 50, Columns: 2]\n",
       "  Relationships:\n",
       "    recordings.engine_no -> engines.engine_no\n",
       "    recordings.settings_clusters -> settings_clusters.settings_clusters"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "\n",
    "nclusters = 50\n",
    "\n",
    "def make_entityset(data, nclusters, kmeans=None):\n",
    "    X = data[['operational_setting_1', 'operational_setting_2', 'operational_setting_3']]\n",
    "    if kmeans:\n",
    "        kmeans=kmeans\n",
    "    else:\n",
    "        kmeans = KMeans(n_clusters=nclusters).fit(X)\n",
    "    data['settings_clusters'] = kmeans.predict(X)\n",
    "    \n",
    "    es = ft.EntitySet('Dataset')\n",
    "    es.entity_from_dataframe(dataframe=data,\n",
    "                             entity_id='recordings',\n",
    "                             index='index',\n",
    "                             time_index='time')\n",
    "\n",
    "    es.normalize_entity(base_entity_id='recordings', \n",
    "                        new_entity_id='engines',\n",
    "                        index='engine_no')\n",
    "    \n",
    "    es.normalize_entity(base_entity_id='recordings', \n",
    "                        new_entity_id='settings_clusters',\n",
    "                        index='settings_clusters')\n",
    "    \n",
    "    return es, kmeans\n",
    "es, kmeans = make_entityset(data, nclusters)\n",
    "es"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize EntitySet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: Dataset Pages: 1 -->\n",
       "<svg width=\"556pt\" height=\"573pt\"\n",
       " viewBox=\"0.00 0.00 556.00 573.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 569)\">\n",
       "<title>Dataset</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-569 552,-569 552,4 -4,4\"/>\n",
       "<!-- recordings -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>recordings</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"167.5,-98.5 167.5,-564.5 379.5,-564.5 379.5,-98.5 167.5,-98.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"273.5\" y=\"-549.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">recordings</text>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"167.5,-541.5 379.5,-541.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-526.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">index : index</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-511.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">engine_no : id</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-496.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">time_in_cycles : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-481.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">operational_setting_1 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-466.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">operational_setting_2 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-451.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">operational_setting_3 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-436.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_1 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-421.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_2 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-406.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_3 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-391.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_4 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-376.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_5 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-361.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_6 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-346.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_7 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-331.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_8 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-316.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_9 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-301.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_10 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-286.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_11 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-271.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_12 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-256.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_13 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-241.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_14 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-226.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_15 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-211.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_16 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-196.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_17 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-181.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_18 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-166.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_19 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-151.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_20 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-136.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sensor_measurement_21 : numeric</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-121.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">time : datetime_time_index</text>\n",
       "<text text-anchor=\"start\" x=\"175.5\" y=\"-106.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">settings_clusters : id</text>\n",
       "</g>\n",
       "<!-- engines -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>engines</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"0,-.5 0,-61.5 265,-61.5 265,-.5 0,-.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"132.5\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">engines</text>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"0,-38.5 265,-38.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-23.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">engine_no : index</text>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-8.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">first_recordings_time : datetime_time_index</text>\n",
       "</g>\n",
       "<!-- recordings&#45;&gt;engines -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>recordings&#45;&gt;engines</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M216.25,-98.4381C216.25,-98.4381 216.25,-71.5741 216.25,-71.5741\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"219.7501,-71.574 216.25,-61.5741 212.7501,-71.5741 219.7501,-71.574\"/>\n",
       "<text text-anchor=\"middle\" x=\"186.75\" y=\"-73.8061\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">engine_no</text>\n",
       "</g>\n",
       "<!-- settings_clusters -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>settings_clusters</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"283,-.5 283,-61.5 548,-61.5 548,-.5 283,-.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"415.5\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">settings_clusters</text>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"283,-38.5 548,-38.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"291\" y=\"-23.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">settings_clusters : index</text>\n",
       "<text text-anchor=\"start\" x=\"291\" y=\"-8.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">first_recordings_time : datetime_time_index</text>\n",
       "</g>\n",
       "<!-- recordings&#45;&gt;settings_clusters -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>recordings&#45;&gt;settings_clusters</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M331.25,-98.4381C331.25,-98.4381 331.25,-71.5741 331.25,-71.5741\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"334.7501,-71.574 331.25,-61.5741 327.7501,-71.5741 334.7501,-71.574\"/>\n",
       "<text text-anchor=\"middle\" x=\"284.75\" y=\"-73.8061\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">settings_clusters</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0xa197c8f98>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: DFS and Creating a Model\n",
    "In addition to changing our `EntitySet` structure, we're also going to use the [Complexity](http://tsfresh.readthedocs.io/en/latest/api/tsfresh.feature_extraction.html#tsfresh.feature_extraction.feature_calculators.cid_ce) time series primitive from the package [tsfresh](https://github.com/blue-yonder/tsfresh). Any function that takes in a pandas `Series` and outputs a float can be converted into an aggregation primitive using the `make_agg_primitive` function as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 302 features\n",
      "Elapsed: 02:58 | Remaining: 00:00 | Progress: 100%|██████████| Calculated: 4/4 chunks\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LAST(recordings.time_in_cycles)</th>\n",
       "      <th>LAST(recordings.operational_setting_1)</th>\n",
       "      <th>LAST(recordings.operational_setting_2)</th>\n",
       "      <th>LAST(recordings.operational_setting_3)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_1)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_2)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_3)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_4)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_5)</th>\n",
       "      <th>LAST(recordings.sensor_measurement_6)</th>\n",
       "      <th>...</th>\n",
       "      <th>COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_13))</th>\n",
       "      <th>COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_14))</th>\n",
       "      <th>COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_15))</th>\n",
       "      <th>COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_16))</th>\n",
       "      <th>COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_17))</th>\n",
       "      <th>COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_18))</th>\n",
       "      <th>COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_19))</th>\n",
       "      <th>COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_20))</th>\n",
       "      <th>COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_21))</th>\n",
       "      <th>RUL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>engine_no</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>255</td>\n",
       "      <td>35.0069</td>\n",
       "      <td>0.8400</td>\n",
       "      <td>100.0</td>\n",
       "      <td>449.44</td>\n",
       "      <td>555.72</td>\n",
       "      <td>1353.47</td>\n",
       "      <td>1123.29</td>\n",
       "      <td>5.48</td>\n",
       "      <td>7.97</td>\n",
       "      <td>...</td>\n",
       "      <td>5.189777</td>\n",
       "      <td>138.960656</td>\n",
       "      <td>0.725119</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>36.031453</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.709334</td>\n",
       "      <td>2.289717</td>\n",
       "      <td>66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>229</td>\n",
       "      <td>24.9997</td>\n",
       "      <td>0.6209</td>\n",
       "      <td>60.0</td>\n",
       "      <td>462.54</td>\n",
       "      <td>536.90</td>\n",
       "      <td>1264.76</td>\n",
       "      <td>1054.70</td>\n",
       "      <td>7.05</td>\n",
       "      <td>9.03</td>\n",
       "      <td>...</td>\n",
       "      <td>5.739238</td>\n",
       "      <td>359.581095</td>\n",
       "      <td>1.249528</td>\n",
       "      <td>0.069282</td>\n",
       "      <td>45.819376</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.022037</td>\n",
       "      <td>2.740721</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13</td>\n",
       "      <td>10.0047</td>\n",
       "      <td>0.2512</td>\n",
       "      <td>100.0</td>\n",
       "      <td>489.05</td>\n",
       "      <td>604.45</td>\n",
       "      <td>1489.04</td>\n",
       "      <td>1305.94</td>\n",
       "      <td>10.52</td>\n",
       "      <td>15.48</td>\n",
       "      <td>...</td>\n",
       "      <td>1.931952</td>\n",
       "      <td>144.385849</td>\n",
       "      <td>0.310511</td>\n",
       "      <td>0.024495</td>\n",
       "      <td>12.887374</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.893923</td>\n",
       "      <td>1.144065</td>\n",
       "      <td>294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>218</td>\n",
       "      <td>42.0043</td>\n",
       "      <td>0.8413</td>\n",
       "      <td>100.0</td>\n",
       "      <td>445.00</td>\n",
       "      <td>549.49</td>\n",
       "      <td>1362.89</td>\n",
       "      <td>1133.31</td>\n",
       "      <td>3.91</td>\n",
       "      <td>5.72</td>\n",
       "      <td>...</td>\n",
       "      <td>5.288050</td>\n",
       "      <td>391.499664</td>\n",
       "      <td>1.368005</td>\n",
       "      <td>0.135109</td>\n",
       "      <td>49.847261</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.690072</td>\n",
       "      <td>4.094572</td>\n",
       "      <td>56</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>186</td>\n",
       "      <td>42.0037</td>\n",
       "      <td>0.8416</td>\n",
       "      <td>100.0</td>\n",
       "      <td>445.00</td>\n",
       "      <td>549.76</td>\n",
       "      <td>1372.96</td>\n",
       "      <td>1146.10</td>\n",
       "      <td>3.91</td>\n",
       "      <td>5.72</td>\n",
       "      <td>...</td>\n",
       "      <td>8.086404</td>\n",
       "      <td>530.975717</td>\n",
       "      <td>1.233570</td>\n",
       "      <td>0.181146</td>\n",
       "      <td>48.756334</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.346465</td>\n",
       "      <td>4.327066</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 303 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           LAST(recordings.time_in_cycles)  \\\n",
       "engine_no                                    \n",
       "1                                      255   \n",
       "2                                      229   \n",
       "3                                       13   \n",
       "4                                      218   \n",
       "5                                      186   \n",
       "\n",
       "           LAST(recordings.operational_setting_1)  \\\n",
       "engine_no                                           \n",
       "1                                         35.0069   \n",
       "2                                         24.9997   \n",
       "3                                         10.0047   \n",
       "4                                         42.0043   \n",
       "5                                         42.0037   \n",
       "\n",
       "           LAST(recordings.operational_setting_2)  \\\n",
       "engine_no                                           \n",
       "1                                          0.8400   \n",
       "2                                          0.6209   \n",
       "3                                          0.2512   \n",
       "4                                          0.8413   \n",
       "5                                          0.8416   \n",
       "\n",
       "           LAST(recordings.operational_setting_3)  \\\n",
       "engine_no                                           \n",
       "1                                           100.0   \n",
       "2                                            60.0   \n",
       "3                                           100.0   \n",
       "4                                           100.0   \n",
       "5                                           100.0   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_1)  \\\n",
       "engine_no                                          \n",
       "1                                         449.44   \n",
       "2                                         462.54   \n",
       "3                                         489.05   \n",
       "4                                         445.00   \n",
       "5                                         445.00   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_2)  \\\n",
       "engine_no                                          \n",
       "1                                         555.72   \n",
       "2                                         536.90   \n",
       "3                                         604.45   \n",
       "4                                         549.49   \n",
       "5                                         549.76   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_3)  \\\n",
       "engine_no                                          \n",
       "1                                        1353.47   \n",
       "2                                        1264.76   \n",
       "3                                        1489.04   \n",
       "4                                        1362.89   \n",
       "5                                        1372.96   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_4)  \\\n",
       "engine_no                                          \n",
       "1                                        1123.29   \n",
       "2                                        1054.70   \n",
       "3                                        1305.94   \n",
       "4                                        1133.31   \n",
       "5                                        1146.10   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_5)  \\\n",
       "engine_no                                          \n",
       "1                                           5.48   \n",
       "2                                           7.05   \n",
       "3                                          10.52   \n",
       "4                                           3.91   \n",
       "5                                           3.91   \n",
       "\n",
       "           LAST(recordings.sensor_measurement_6) ...   \\\n",
       "engine_no                                        ...    \n",
       "1                                           7.97 ...    \n",
       "2                                           9.03 ...    \n",
       "3                                          15.48 ...    \n",
       "4                                           5.72 ...    \n",
       "5                                           5.72 ...    \n",
       "\n",
       "           COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_13))  \\\n",
       "engine_no                                                                                          \n",
       "1                                                   5.189777                                       \n",
       "2                                                   5.739238                                       \n",
       "3                                                   1.931952                                       \n",
       "4                                                   5.288050                                       \n",
       "5                                                   8.086404                                       \n",
       "\n",
       "           COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_14))  \\\n",
       "engine_no                                                                                          \n",
       "1                                                 138.960656                                       \n",
       "2                                                 359.581095                                       \n",
       "3                                                 144.385849                                       \n",
       "4                                                 391.499664                                       \n",
       "5                                                 530.975717                                       \n",
       "\n",
       "           COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_15))  \\\n",
       "engine_no                                                                                          \n",
       "1                                                   0.725119                                       \n",
       "2                                                   1.249528                                       \n",
       "3                                                   0.310511                                       \n",
       "4                                                   1.368005                                       \n",
       "5                                                   1.233570                                       \n",
       "\n",
       "           COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_16))  \\\n",
       "engine_no                                                                                          \n",
       "1                                                   0.000000                                       \n",
       "2                                                   0.069282                                       \n",
       "3                                                   0.024495                                       \n",
       "4                                                   0.135109                                       \n",
       "5                                                   0.181146                                       \n",
       "\n",
       "           COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_17))  \\\n",
       "engine_no                                                                                          \n",
       "1                                                  36.031453                                       \n",
       "2                                                  45.819376                                       \n",
       "3                                                  12.887374                                       \n",
       "4                                                  49.847261                                       \n",
       "5                                                  48.756334                                       \n",
       "\n",
       "           COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_18))  \\\n",
       "engine_no                                                                                          \n",
       "1                                                          0                                       \n",
       "2                                                          0                                       \n",
       "3                                                          0                                       \n",
       "4                                                          0                                       \n",
       "5                                                          0                                       \n",
       "\n",
       "           COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_19))  \\\n",
       "engine_no                                                                                          \n",
       "1                                                        0.0                                       \n",
       "2                                                        0.0                                       \n",
       "3                                                        0.0                                       \n",
       "4                                                        0.0                                       \n",
       "5                                                        0.0                                       \n",
       "\n",
       "           COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_20))  \\\n",
       "engine_no                                                                                          \n",
       "1                                                   3.709334                                       \n",
       "2                                                   4.022037                                       \n",
       "3                                                   1.893923                                       \n",
       "4                                                   5.690072                                       \n",
       "5                                                   6.346465                                       \n",
       "\n",
       "           COMPLEXITY(recordings.settings_clusters.COMPLEXITY(recordings.sensor_measurement_21))  \\\n",
       "engine_no                                                                                          \n",
       "1                                                   2.289717                                       \n",
       "2                                                   2.740721                                       \n",
       "3                                                   1.144065                                       \n",
       "4                                                   4.094572                                       \n",
       "5                                                   4.327066                                       \n",
       "\n",
       "           RUL  \n",
       "engine_no       \n",
       "1           66  \n",
       "2           70  \n",
       "3          294  \n",
       "4           56  \n",
       "5            7  \n",
       "\n",
       "[5 rows x 303 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from featuretools.primitives import make_agg_primitive\n",
    "import featuretools.variable_types as vtypes\n",
    "\n",
    "from tsfresh.feature_extraction.feature_calculators import (number_peaks, mean_abs_change, \n",
    "                                                            cid_ce, last_location_of_maximum, length)\n",
    "\n",
    "\n",
    "Complexity = make_agg_primitive(lambda x: cid_ce(x, False),\n",
    "                              input_types=[vtypes.Numeric],\n",
    "                              return_type=vtypes.Numeric,\n",
    "                              name=\"complexity\")\n",
    "\n",
    "fm, features = ft.dfs(entityset=es, \n",
    "                      target_entity='engines',\n",
    "                      agg_primitives=['last', 'max', Complexity],\n",
    "                      trans_primitives=[],\n",
    "                      chunk_size=.26,\n",
    "                      cutoff_time=cutoff_time_list[0],\n",
    "                      max_depth=3,\n",
    "                      verbose=True)\n",
    "\n",
    "fm.to_csv('advanced_fm.csv')\n",
    "fm.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We build 4 more feature matrices with the same feature set but different cutoff times. That lets us test the pipeline multiple times before using it on test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [11:47<00:00, 177.18s/it]\n"
     ]
    }
   ],
   "source": [
    "fm_list = [fm]\n",
    "for i in tqdm(range(1, splits)):\n",
    "    fm = ft.calculate_feature_matrix(entityset=make_entityset(data, nclusters, kmeans=kmeans)[0], \n",
    "                                     features=features, \n",
    "                                     chunk_size=.26, \n",
    "                                     cutoff_time=cutoff_time_list[i])\n",
    "    fm_list.append(fm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[40.2, 39.7, 35.2, 44.5, 40.8]\n",
      "Average MAE: 40.1, Std: 2.98\n",
      "\n",
      "1: MAX(recordings.settings_clusters.LAST(recordings.sensor_measurement_13)) [0.105]\n",
      "2: MAX(recordings.sensor_measurement_11) [0.074]\n",
      "3: MAX(recordings.sensor_measurement_13) [0.067]\n",
      "4: MAX(recordings.sensor_measurement_4) [0.047]\n",
      "5: MAX(recordings.settings_clusters.LAST(recordings.sensor_measurement_11)) [0.045]\n",
      "-----\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.feature_selection import RFE\n",
    "def pipeline_for_test(fm_list, hyperparams={'n_estimators':100, 'max_feats':50, 'nfeats':50}, do_selection=False):\n",
    "    scores = []\n",
    "    regs = []\n",
    "    selectors = []\n",
    "    for fm in fm_list:\n",
    "        X = fm.copy().fillna(0)\n",
    "        y = X.pop('RUL')\n",
    "        reg = RandomForestRegressor(n_estimators=int(hyperparams['n_estimators']), \n",
    "                                    max_features=min(int(hyperparams['max_feats']), \n",
    "                                                     int(hyperparams['nfeats'])))\n",
    "        X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
    "        if do_selection:\n",
    "            reg2 = RandomForestRegressor(n_estimators=10, n_jobs=3)\n",
    "            selector = RFE(reg2, int(hyperparams['nfeats']), step=25)\n",
    "            selector.fit(X_train, y_train)\n",
    "            X_train = selector.transform(X_train)\n",
    "            X_test = selector.transform(X_test)\n",
    "            selectors.append(selector)\n",
    "        reg.fit(X_train, y_train)\n",
    "        regs.append(reg)\n",
    "        \n",
    "        preds = reg.predict(X_test)\n",
    "        scores.append(mean_absolute_error(preds, y_test))\n",
    "    return scores, regs, selectors    \n",
    "scores, regs, selectors = pipeline_for_test(fm_list)\n",
    "print([float('{:.1f}'.format(score)) for score in scores])\n",
    "print('Average MAE: {:.1f}, Std: {:.2f}\\n'.format(np.mean(scores), np.std(scores)))\n",
    "\n",
    "most_imp_feats = utils.feature_importances(fm_list[0], regs[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded data with:\n",
      "41214 Recordings\n",
      "248 Engines\n",
      "21 Sensor Measurements\n",
      "3 Operational Settings\n",
      "Elapsed: 00:02 | Remaining: 00:00 | Progress: 100%|██████████| Calculated: 1/1 chunks\n",
      "Mean Abs Error: 28.59\n"
     ]
    }
   ],
   "source": [
    "data_test = utils.load_data('data/test_FD004.txt')\n",
    "\n",
    "es_test, _ = make_entityset(data_test, nclusters, kmeans=kmeans)\n",
    "fm_test = ft.calculate_feature_matrix(entityset=es_test, features=features, verbose=True, chunk_size='cutoff time')\n",
    "X = fm_test.copy().fillna(0)\n",
    "y = pd.read_csv('data/RUL_FD004.txt', sep=' ', header=-1, names=['RUL'], index_col=False)\n",
    "preds = regs[0].predict(X)\n",
    "print('Mean Abs Error: {:.2f}'.format(mean_absolute_error(preds, y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Feature Selection and Scoring\n",
    "Here, we'll use [Recursive Feature Elimination](http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html). In order to set ourselves up for later optimization, we're going to write a generic `pipeline` function which takes in a set of hyperparameters and returns a score. Our pipeline will first run `RFE` and then split the remaining data for scoring by a `RandomForestRegressor`. We're going to pass in a list of hyperparameters, which we will tune later. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can use that selector and regressor to score the test values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: Hyperparameter Tuning\n",
    "Because of the way we set up our pipeline, we can use a Gaussian Process to tune the hyperparameters. We will use [BTB](https://github.com/HDI-Project/BTB) from the [HDI Project](https://github.com/HDI-Project). This will search through the hyperparameters `n_estimators` and `max_feats` for RandomForest, and the number of features for RFE to find the hyperparameter set that has the best average score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/30 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[n_est, max_feats, nfeats]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  3%|▎         | 1/30 [00:13<06:45, 13.99s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0. {'n_estimators': 80, 'max_feats': 20, 'nfeats': 40} -- Average MAE: 37.1, Std: 2.28\n",
      "Raw: [40.0, 38.0, 34.1, 34.8, 38.5]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 33%|███▎      | 10/30 [02:23<04:40, 14.02s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9. {'n_estimators': 80, 'max_feats': 21, 'nfeats': 41} -- Average MAE: 32.9, Std: 4.15\n",
      "Raw: [34.4, 38.5, 26.3, 30.5, 34.7]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 30/30 [07:09<00:00, 14.59s/it]\n"
     ]
    }
   ],
   "source": [
    "from btb import HyperParameter, ParamTypes\n",
    "from btb.tuning import GP\n",
    "\n",
    "def run_btb(fm_list, n=30):\n",
    "    hyperparam_ranges = [\n",
    "            ('n_estimators', HyperParameter(ParamTypes.INT, [10, 200])),\n",
    "            ('max_feats', HyperParameter(ParamTypes.INT, [5, 50])),\n",
    "            ('nfeats', HyperParameter(ParamTypes.INT, [10, 70])),\n",
    "    ]\n",
    "    tuner = GP(hyperparam_ranges)\n",
    "\n",
    "    tested_parameters = np.zeros((n, len(hyperparam_ranges)), dtype=object)\n",
    "    scores = []\n",
    "    \n",
    "    print('[n_est, max_feats, nfeats]')\n",
    "    best = 45\n",
    "\n",
    "    for i in tqdm(range(n)):\n",
    "        hyperparams = tuner.propose()\n",
    "        cvscores, regs, selectors = pipeline_for_test(fm_list, hyperparams=hyperparams, do_selection=True)\n",
    "        bound = np.mean(cvscores)\n",
    "        tested_parameters[i, :] = hyperparams\n",
    "        tuner.add(hyperparams, -np.mean(cvscores))\n",
    "        if np.mean(cvscores) + np.std(cvscores) < best:\n",
    "            best = np.mean(cvscores)\n",
    "            best_hyperparams = hyperparams\n",
    "            best_reg = regs[0]\n",
    "            best_sel = selectors[0]\n",
    "            print('{}. {} -- Average MAE: {:.1f}, Std: {:.2f}'.format(i, \n",
    "                                                                      best_hyperparams, \n",
    "                                                                      np.mean(cvscores), \n",
    "                                                                      np.std(cvscores)))\n",
    "            print('Raw: {}'.format([float('{:.1f}'.format(s)) for s in cvscores]))\n",
    "\n",
    "    return best_hyperparams, (best_sel, best_reg)\n",
    "\n",
    "best_hyperparams, best_pipeline = run_btb(fm_list, n=30)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Abs Error on Test: 29.48\n",
      "1: MAX(recordings.sensor_measurement_13) [0.139]\n",
      "2: MAX(recordings.settings_clusters.LAST(recordings.sensor_measurement_13)) [0.104]\n",
      "3: MAX(recordings.sensor_measurement_11) [0.084]\n",
      "4: MAX(recordings.settings_clusters.LAST(recordings.sensor_measurement_11)) [0.083]\n",
      "5: COMPLEXITY(recordings.settings_clusters.LAST(recordings.sensor_measurement_8)) [0.071]\n",
      "-----\n",
      "\n"
     ]
    }
   ],
   "source": [
    "X = fm_test.copy().fillna(0)\n",
    "y = pd.read_csv('data/RUL_FD004.txt', sep=' ', header=-1, names=['RUL'], index_col=False)\n",
    "\n",
    "preds = best_pipeline[1].predict(best_pipeline[0].transform(X))\n",
    "score = mean_absolute_error(preds, y)\n",
    "print('Mean Abs Error on Test: {:.2f}'.format(score))\n",
    "most_imp_feats = utils.feature_importances(X.iloc[:, best_pipeline[0].support_], best_pipeline[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix: Averaging old scores\n",
    "To make a fair comparison between the previous notebook and this one, we should average scores where possible. The work in this section is exactly the work in the previous notebook plus some code for taking the average in the validation step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 302 features\n",
      "Elapsed: 03:16 | Remaining: 00:00 | Progress: 100%|██████████| Calculated: 11/11 chunks"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/4 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [13:36<00:00, 205.04s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[30.0, 41.2, 43.3, 34.9, 37.4]\n",
      "Average MAE: 37.35, Std: 4.72\n",
      "\n",
      "[67.4, 64.3, 63.4, 65.1, 61.0]\n",
      "Baseline by Median MAE: 64.23, Std: 2.10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from featuretools.primitives import Min\n",
    "old_fm, features = ft.dfs(entityset=es, \n",
    "                      target_entity='engines',\n",
    "                      agg_primitives=['last', 'max', 'min'],\n",
    "                      trans_primitives=[],\n",
    "                      cutoff_time=cutoff_time_list[0],\n",
    "                      max_depth=3,\n",
    "                      verbose=True)\n",
    "\n",
    "old_fm_list = [old_fm]\n",
    "for i in tqdm(range(1, splits)):\n",
    "    old_fm = ft.calculate_feature_matrix(entityset=make_entityset(data, nclusters, kmeans=kmeans)[0], \n",
    "                                     features=features, \n",
    "                                     cutoff_time=cutoff_time_list[i])\n",
    "    old_fm_list.append(fm)\n",
    "\n",
    "old_scores = []\n",
    "median_scores = []\n",
    "for fm in old_fm_list:\n",
    "    X = fm.copy().fillna(0)\n",
    "    y = X.pop('RUL')\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
    "    reg = RandomForestRegressor(n_estimators=10)\n",
    "    reg.fit(X_train, y_train)\n",
    "    preds = reg.predict(X_test)\n",
    "    old_scores.append(mean_absolute_error(preds, y_test))\n",
    "    \n",
    "    medianpredict = [np.median(y_train) for _ in y_test]\n",
    "    median_scores.append(mean_absolute_error(medianpredict, y_test))\n",
    "\n",
    "print([float('{:.1f}'.format(score)) for score in old_scores])\n",
    "print('Average MAE: {:.2f}, Std: {:.2f}\\n'.format(np.mean(old_scores), np.std(old_scores)))\n",
    "\n",
    "print([float('{:.1f}'.format(score)) for score in median_scores])\n",
    "print('Baseline by Median MAE: {:.2f}, Std: {:.2f}\\n'.format(np.mean(median_scores), np.std(median_scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[49.5, 52.8, 49.0, 49.5, 48.3]\n",
      "Baseline by Median MAE: 49.82, Std: 1.58\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y = pd.read_csv('data/RUL_FD004.txt', sep=' ', header=-1, names=['RUL'], index_col=False)\n",
    "median_scores_2 = []\n",
    "for ct in cutoff_time_list:\n",
    "    medianpredict2 = [np.median(ct['RUL'].values) for _ in y.values]\n",
    "    median_scores_2.append(mean_absolute_error(medianpredict2, y))\n",
    "print([float('{:.1f}'.format(score)) for score in median_scores_2])\n",
    "print('Baseline by Median MAE: {:.2f}, Std: {:.2f}\\n'.format(np.mean(median_scores_2), np.std(median_scores_2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save output files\n",
    "\n",
    "import os\n",
    "\n",
    "try:\n",
    "    os.mkdir(\"output\")\n",
    "except:\n",
    "    pass\n",
    "\n",
    "fm.to_csv('output/advanced_train_feature_matrix.csv')\n",
    "cutoff_time_list[0].to_csv('output/advanced_train_label_times.csv')\n",
    "fm_test.to_csv('output/advanced_test_feature_matrix.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>\n",
    "    <img src=\"https://www.featurelabs.com/wp-content/uploads/2017/12/logo.png\" alt=\"Featuretools\" />\n",
    "</p>\n",
    "\n",
    "Featuretools was created by the developers at [Feature Labs](https://www.featurelabs.com/). If building impactful data science pipelines is important to you or your business, please [get in touch](https://www.featurelabs.com/contact)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
